Image motion display method and apparatus

ABSTRACT

In a display apparatus connected to an image processing apparatus which detects the change amount in an input image and stores, as a detected image, an image during a period with a change amount equal to or more than a predetermined amount, the change log of the change amount and the detected image are received, the change log is displayed by using a graph, and information related to the detected image is displayed on the graph in a superposed manner.

CROSS REFERENCE TO RELATED APPLICATION

This application is a Divisional of U.S. application Ser. No.10/932,001, filed on Sep. 2, 2004, the entire disclosure of which isincorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to a display apparatus, image processingapparatus, and image processing system and, more particularly, to atechnique for storing an image when an image change amount not less thana predetermined amount is detected in a monitoring system or cameraserver.

The present invention also relates to an image processing apparatus andmethod for an image photographed by a camera and, more particularly, tosetting of motion detection sensitivity.

The present invention also relates to an image motion detection methodand apparatus for detecting motion of a moving image, and a program andstorage medium thereof.

BACKGROUND OF THE INVENTION

In monitoring systems which photograph a moving image by using a camera,a technique for automatically detecting a change (scene change) in aphotographed moving image by image processing and saving/accumulating(storing) the image when the change is detected has been put intopractical use.

In such systems, generally, a plurality of images which are photographedand accumulated when changes are detected are laid out and displayed inthe chronological order together with the photography times.

In addition, a function of browsing live images or images photographedat the time of change detection via a network and executing variouskinds of setting including parameters of change detection from a remotesite has also been put into practical use.

On the other hand, in a video editing system, a change in scene isdetected by image processing, and the scene change time and an image atthat time are visually displayed on an editing window.

For example, Japanese Patent Laid-Open No. 11-32301 discloses an imageprocessing apparatus including a GUI (Graphical User Interface) whichautomatically executes cut detection in video editing software anddisplays the start image of the same scene.

Japanese Patent Laid-Open No. 9-65287 discloses a moving image featurescene detection apparatus which obtains the same scene on the basis ofthe color tone, speaker, subtitle, and the like and displays therepresentative image in the scene.

In the above-described monitoring systems, images at the time of changedetection can be displayed in a list in the time-series order. However,the magnitude of the change amount between the images is unknown.

On the other hand, in the video editing systems disclosed in the twopatent references, the change time and the first scene or representativeimage by cut detection are displayed. In this case as well, themagnitude of the change amount between the images is unknown.

If the magnitude of the change amount between the displayed images isunknown, as described above, it is difficult to find the portion withthe largest change amount from accumulated images.

In the above-described systems, the parameters may be changed such thatonly images with large change amounts are accumulated and displayed. Inthis case, however, an image whose change amount is smaller than thethreshold value is not accumulated. Hence, it may be impossible at ahigh probability to see desired images after image accumulation.

In monitoring systems and, more particularly, monitoring systems whichcan execute remote monitoring using computer networks, a function calledmotion detection, change detection, or dynamic detection is essential.These detection functions will be referred to as “motion detection”hereinafter (e.g., Japanese Patent Laid-Open No. 08-297792).

In “motion detection”, a change in image between frames or the motion ofan object in image between frames sensed by a video camera is detectedby image processing. The user is notified of the detection result on adisplay device or through an output device such as a loudspeaker.

In use of the motion detection function, it is important to determinethe threshold value (sensitivity setting) to set the degree of change inimage between frames or the degree of motion of the object in imagebetween frames, which is the criterion for notification to the user. Ifthe detection sensitivity is too low, no invaders can be detected.Conversely, if the sensitivity setting is too high, notification takesplace even when it is unnecessary.

In some of the conventional systems, sensitivity setting is so easy thatone of several preset levels from low to high sensitivity is selected.Alternatively, it is necessary to designate various parameters asnumerical values.

In systems of former type, optimum setting is not always possible inactual operation. In systems of latter type, intuitive setting isimpossible, and designation of appropriate set values is difficult initself.

As described above, in the monitoring systems, the function ofautomatically obtaining a change in monitoring image by image processingand, upon detecting a change, notifying the user of it orsaving/accumulating the image has been put into practical use. Somesystems have a function of displaying an image change amount during theprogress of motion detection processing and interactively adjusting thesensitivity to set the motion detection sensitivity.

For example, a system in which the user can adjust the detectionparameter while observing an indicator that indicates the image changeamount is known. In addition, Japanese Patent Laid-Open Nos. 11-032301and 9-65287 disclose video editing systems which have a function ofobtaining a scene change by image processing and visually displaying thescene change time and the image at that time on an editing window.

On the other hand, Japanese Patent Laid-Open No. 9-200768 discloses amethod of detecting the motion of an image on the basis of thebackground difference or inter-frame difference. In this method, animage with a motion and an image without any motion are input to set anoptimum sensitivity that should prevent any detection error or detectionmiss.

With the conventional moving image motion detection function, thedetection sensitivity can be set. However, the range of actuallysettable values is often wide, and the detection sensitivity must beadjusted by trial and error. The video editing systems have the functionof displaying the scene start image or representative image by cutdetection. However, they have no detection intensity adjustment functionin general. In the invention disclosed in Japanese Patent Laid-Open No.9-200768, an appropriate sensitivity range is presented on the basis ofan image with a motion and an image without any motion, thereby reducingthe load on the user in setting the sensitivity. However, patentreference 3 discloses no specific interface for input by the user.

SUMMARY OF THE INVENTION

It is the first object of the present invention to make it possible todisplay time progress of the change amount in an image and display animage when a change equal to or more than a predetermined amount isdetected in association with the time progress of the change amount.

A display apparatus according to one aspect of the present invention,which achieves the first object, is a display apparatus which isconnected to an image processing apparatus which detects a change amountin an input image and stores, as a detected image, an image obtainedduring a period with a change amount not less than a predeterminedamount, comprising: a reception device which receives a change log ofthe change amount and the detected image, wherein the change log isdisplayed by using a graph, and information related to the detectedimage is displayed on the graph in a superposed manner.

An image processing apparatus according to another aspect of the presentinvention, which achieves the first object, is an image processingapparatus comprising: an encoding device which encodes input image dataat a predetermined time interval; a change detection device whichdetects on the basis of a difference between the image data at the timeinterval whether a change amount in image data exceeds a predeterminedamount; and a storage device which stores a change log representing timeprogress of the change amount and, when the detection device detectsthat the change amount exceeds the predetermined amount, storescorresponding encoded image data as a detected image.

More specifically, in the present invention, in a display apparatusconnected to an image processing apparatus which detects the changeamount in an input image and stores, as a detected image, an imageduring a period with a change amount equal to or more than apredetermined amount, the change log (history) of the change amount andthe detected image are received, the change log is displayed by using agraph, and information related to the detected image is displayed on thegraph in a superposed manner.

Accordingly, the time progress of a change amount in image data during apredetermined period can be displayed by using, e.g., a graph, andinformation related to a detected image can be displayed in associationwith the graph. The time progress of image data can be displayed in avisually convenient form, and a desired detected image can easily bedisplayed.

Further, the first object of the present invention is also achieved by adisplay apparatus which is connected to an image processing apparatusand displays a change log and a detected image, the image processingapparatus comprising an encoding device which encodes input image dataat a predetermined time interval, a change detection device whichdetects on the basis of a difference between the image data at the timeinterval whether a change amount in image data exceeds a predeterminedamount, and a storage device which stores the change log representingtime progress of the change amount and, when the detection devicedetects that the change amount exceeds the predetermined amount, storescorresponding encoded image data as the detected image, comprising: adecoding device which decodes the encoded image data; and a displaydevice which displays the change log by using a graph and displaysinformation related to the detected image on the graph in a superposedmanner.

Moreover, the first object of the present invention is also achieved bya display apparatus which is connected to an image processing apparatusand displays a change log and a detected image, the image processingapparatus comprising an encoding device which encodes input image dataat a predetermined time interval, a change detection device whichdetects on the basis of a difference between the image data at the timeinterval whether a change amount in image data exceeds a predeterminedamount, and a storage device which stores the change log representingtime progress of the change amount and, when the detection devicedetects that the change amount exceeds the predetermined amount, storescorresponding encoded image data as the detected image, the changedetection device detecting the change amount in each of a plurality ofpreset regions of the image data, and the storage device storing thechange log and the detected image for each region, comprising: adecoding device which decodes the encoded image data; an operationdevice which is operated by a user to input one of a selection and aninstruction; and a display device which displays the change log by usinga graph for one of the plurality of regions, which is selected by theoperation device, and displays information related to the detected imagefor each of the plurality of regions on the graph in a superposedmanner.

The above object is also achieved by a display method and imageprocessing method corresponding to the display apparatus and imageprocessing apparatus, a program which implements these methods on acomputer apparatus, and a storage medium which stores the program.

It is the second object of the present invention to provide a techniquefor easily setting a suitable set value which should be used to notify auser that a change between frame images is equal to or more than apredetermined value.

An image processing apparatus according to the present invention, whichachieves the second object, is an image processing apparatus whichobtains a set value to execute notification processing when a differencebetween frame images contained in a moving image is not less than theset value, comprising: an average image generation device which dividesthe frames into groups each including a predetermined number of framesand executes, for each group, processing for generating an average imageof the frame images in the group; a designation device which designatesa start frame and an end frame of the frames; a first calculation devicewhich obtains a maximum difference value and a minimum difference valueof differences between the frame images from the start frame to the endframe and an image obtained by the average image generation device for agroup including the start frame; a second calculation device whichobtains a maximum difference value and a minimum difference value ofdifferences between the frame images from the start frame to the endframe and an image obtained by the average image generation device for agroup including the end frame; and a determination device whichdetermines the set value by using a larger one of the maximum differencevalues calculated by the first calculation device and the secondcalculation device and a smaller one of the minimum difference valuescalculated by the first calculation device and the second calculationdevice.

Another image processing apparatus according to the present invention,which achieves the second object, is an image processing apparatus whichobtains a set value to execute notification processing when a differencebetween frame images contained in a moving image is not less than theset value, comprising: a first average image generation device whichdivides the frames into groups each including a predetermined number offrames and executes, for each group, processing for generating anaverage image of the frame images in the group; a designation devicewhich designates a start frame and an end frame of the frames; a secondaverage image generation device which generates an average image of animage obtained by the first average image generation device for a groupincluding the start frame and an image obtained by the first averageimage generation device for a group including the end frame; a firstspecifying device which specifies a maximum difference value ofdifferences between the frame images from the start frame to the endframe and the image generated by the second average image generationdevice; a second specifying device which specifies a minimum differencevalue of differences between the frame images from the start frame tothe end frame and the image generated by the second average imagegeneration device; and a determination device which determines the setvalue by using the difference value specified by the first specifyingdevice and the difference value specified by the second specifyingdevice.

According to the above arrangement, a set value which should be used tonotify a user that a change between frame images is equal to or morethan a predetermined value can easily be set to a suitable value.

The above object is also achieved by an image processing methodcorresponding to the image processing apparatus, a program whichimplements the method on a computer apparatus, and a storage mediumwhich stores the program.

It is the third object of the present invention to make it possible todesignate a period in which motion is present in a moving image and aperiod in which no motion is present by user operation in obtaining athreshold value setting range useful for appropriate sensitivityadjustment from these periods.

An image motion detection apparatus according to the present invention,which achieves the third object, is an image motion detection apparatuscomprising: an image display device which displays a moving image; aperiod designation device which designates, in accordance with useroperation, a first period in which motion is present in the moving imagedisplayed by the image display device and a second period in which nomotion is present; an upper/lower limit value determination device whichdetermines an upper limit value and a lower limit value of a thresholdvalue on the basis of a change amount in the moving image during thefirst period with respect to a predetermined reference image and achange amount in the moving image during the second period with respectto the predetermined reference image; a threshold value determinationdevice which determines an arbitrary threshold value in accordance withuser operation within a range from the upper limit value to the lowerlimit value of the threshold value, which are determined by theupper/lower limit value determination device; and a motion detectiondevice which detects motion in the moving image on the basis of thethreshold value determined by the threshold value determination deviceand the change amount in the moving image with respect to thepredetermined reference image.

Another image motion detection apparatus according to the presentinvention, which achieves the third object, is an image motion detectionapparatus comprising: an image display device which displays a movingimage; a start instruction device which instructs a start of automaticsetting of sensitivity; and a threshold value determination device whichobtains, on the basis of a plurality of frames, a change amount in themoving image with respect to a predetermined reference image in a statein which the image is assumed to have no motion in response to theinstruction of automatic setting by the start instruction device anddetermines a threshold value for motion detection from the obtainedchange amount.

According to the above arrangement, a first period in which motion ispresent in a moving image and a second period in which no motion ispresent are designated in accordance with user operation, and the upperand lower limit values of a threshold value are determined on the basisof a change amount in the moving image during the first period withrespect to a predetermined reference image and a change amount in themoving image during the second period with respect to the predeterminedreference image. Hence, when obtaining a threshold value setting rangeuseful for appropriate sensitivity adjustment from a period in whichmotion is present in a moving image and a period in which no motion ispresent, these periods can be designated by user operation.

In addition, a start of automatic setting of sensitivity is instructed,a change amount in a moving image with respect to a predeterminedreference image is photographed for a plurality of frames after theinstruction of automatic setting start assuming that no motion ispresent in the image, and a threshold value for motion detection isdetermined from the obtained change amount. Hence, a threshold valueuseful for appropriate sensitivity adjustment can automatically be setby only simple user operation.

The above object is also achieved by an image motion detection methodcorresponding to the image motion detection apparatus, a program whichimplements the method on a computer apparatus, and a storage mediumwhich stores the program.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the hardware configuration of an imagechange detection system according to the first embodiment;

FIG. 2 is a functional block diagram of a camera server shown in FIG. 1;

FIG. 3 is a functional block diagram of a setting client shown in FIG.1;

FIG. 4 is a view showing an example of a window displayed on the displayoperation unit of the setting client;

FIG. 5 is a view showing an example of a window displayed by a changedetection setting unit to execute various kinds of setting;

FIG. 6 is a view showing an example of a window which displays adetection log according to the first embodiment;

FIG. 7 is a flowchart showing the processing procedures of the mainprocess of the camera server;

FIG. 8 is a flowchart showing the processing procedures of the subprocess of the camera server;

FIG. 9 is a flowchart showing the processing procedures of the settingclient;

FIG. 10 is a view showing an example of a window which displays adetection log according to the second embodiment;

FIG. 11 is a view showing an example of a window which displays adetection log according to the third embodiment;

FIG. 12 is a view showing an example of a window which displays adetection log according to the fourth embodiment;

FIG. 13 is a view showing the basic arrangement of an image processingsystem according to the fifth embodiment of the present invention;

FIG. 14 is a view showing a display example of a GUI to notify a userthat a change equal to or more than a set value (to be described later)is detected between frame images input from a video camera 2101;

FIG. 15 is a flowchart of processing for obtaining a set value;

FIG. 16 is a view showing a display example of a GUI which displays alist of frame images and executes an operation of setting a set value;

FIG. 17 is a view for explaining average image generation processing;

FIG. 18 is a view for explaining time code information;

FIG. 19 is a view for explaining the outline of motion detectionprocessing by a difference;

FIGS. 20A and 20B are graphs showing a change in change amount overtime;

FIGS. 21A and 21B are views showing a GUI for motion detectionsensitivity setting according to the eighth embodiment of the presentinvention;

FIGS. 22A and 22B are flowcharts showing the operation procedures of amotion detection setting program according to the eighth embodiment ofthe present invention;

FIG. 23 is a flowchart showing the operation procedures of the motiondetection setting program according to the eighth embodiment of thepresent invention;

FIG. 24 is a view showing a GUI for motion detection sensitivity settingaccording to the ninth embodiment of the present invention;

FIG. 25 is a flowchart showing the operation procedures of a motiondetection sensitivity setting program according to the ninth embodimentof the present invention;

FIG. 26 is a flowchart showing the operation procedures of the motiondetection sensitivity setting program according to the ninth embodimentof the present invention;

FIG. 27 is a view for explaining a method of automatically setting amotion detection target region according to the 10th embodiment of thepresent invention;

FIG. 28 is a view showing a GUI for motion detection setting accordingto the 10th embodiment of the present invention;

FIG. 29 is a view showing a GUI for motion detection setting accordingto the 11th embodiment of the present invention; and

FIG. 30 is a flowchart showing the operation procedures of a motiondetection setting program according to the 11th embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

In this specification, “accumulate” is used in the same sense as “store”or “save” and means that information is stored in a playback enablestate.

First Embodiment

In the first embodiment, an image change detection system will bedescribed, in which a change in photographed image is detected in acamera server, and a graph representing a time-series change inconsequently accumulated change detection amount and the image at thetime of change detection are displayed on the screen of an image changedetection setting client (to be simply referred to as a setting clienthereinafter).

FIG. 1 is a block diagram showing the hardware configuration of theimage change detection system according to this embodiment. The imagechange detection system according to this embodiment includes a cameraserver 101 and a setting client 114, which are connected through anetwork 115.

The camera server 101 acquires an image photographed by a camera 113 andtransmits the image to the setting client 114 through the network 115.In addition, image change detection processing is executed in the cameraserver and accumulates the image at the time of change detection and thelog (history) of change detection amounts. The camera 113 may photographeither a still image or a moving image.

Software which displays a received video image, software which sets thechange detection function, and software which displays the change logand the image accumulated at the time of change detection run on thesetting client 114. The setting client 114 transmits a set value set bythe setting software to the camera server 101. The camera server 101receives the set value, and from this time, executes change detectionprocessing on the basis of the set value.

The camera server 101 comprises a CPU 102, RAM 103, ROM 104, andsecondary storage device 105. The camera server 101 also comprises avideo RAM (VRAM) 106 for screen display. A monitor 107 is connected tothe camera server 101. The camera server 101 comprises a peripheraldevice interface 108 to connect peripheral devices. The peripheraldevice interface 108 is connected to a keyboard 110 to be operated bythe user, a pointing device 111 such a mouse, and a camera 113 with orwithout a pan head. The camera 113 is connected to the peripheral deviceinterface 108 which transmits/receives a digital signal or a videocapture card 112 which receives a video signal. The camera server 101also comprises a network interface 109 for connection to the network115. As the standard of the peripheral device interface 108, variousstandards such as PS/2, RS-232C, USB, and IEEE1394 are present. However,this embodiment does not depend on these standards.

The CPU 102, RAM 103, ROM 104, secondary storage device 105, VRAM 106,peripheral device interface 108, network interface 109, and videocapture card 112 are connected to the internal bus.

The above-described arrangement of the camera server 101 can easily beimplemented by using a commercially available personal computer andcamera. The server main body can also be operated from an externaldevice through the network. Hence, a so-called set-top box structurehaving none of the VRAM 106, monitor 107, keyboard 110, and mouse 111can also be employed without any problem.

The setting client 114 also has almost the same arrangement as thecamera server 101 except input devices connected. More specifically, thesetting client 114 includes the CPU 102, RAM 103, ROM 104, secondarystorage device 105, VRAM 106, monitor 107, peripheral device interface108, network interface 109, keyboard 110, and pointing device (mouse)111.

The setting client 114 can also easily be implemented by using acommercially available personal computer. The already described softwareprograms which execute video display, change detection setting, andchange log display are stored in the secondary storage device 105. A GUIis displayed on the monitor 107 by application software so that the userinteractively executes the operation by using the keyboard 110 and mouse111.

The setting client 114 acquires an image from the camera server 101 inthe following way. For example, when a URL-encoded command istransmitted by HTTP, images corresponding to a plurality of frames arereturned. This scheme is a known technique used by a network camera onthe market.

In this embodiment, the Internet based on the IP protocol is assumed asthe network 115. However, the implementation scheme is not limited, andany other transmission path which can transmit/receive a digital signaland has a sufficient capacity for image communication can be used.

Examples of the functional blocks of the camera server will be describednext with reference to FIG. 2. Only parts related to image processingwill be described with reference to FIG. 2, and a description of partsrelated to camera control and system control will be omitted.

Referring to FIG. 2, an image signal input from a camera 201 isconverted into a digital signal by an A/D conversion unit 202 andencoded by an encoding unit 203. JPEG or MPEG is used as the encodingscheme. However, this embodiment does not depend on the encoding scheme.Next, a change detection unit 205 executes change detection processingof the image. The change amount log and the accumulated image at thetime of change detection by the change detection unit 205 are saved andmanaged by a change log accumulation unit 207. The encoded image, thechange detection processing result, the change log, and the accumulatedimage at the time of change detection are transmitted from atransmission unit 204 in response to a request from the setting client.A reception unit 206 receives encoding or change detection settings, alive image request, a change log request, or an accumulated imagerequest transmitted from the setting client.

A change detection processing scheme using the inter-frame difference orbackground difference is known. As the feature amount of the inter-framedifference, the absolute value of the lightness difference betweenpixels at the same coordinates or the absolute value of the differencein DCT coefficient in a JPEG coded block is used. When a value obtainedby integrating these absolute values in the entire image is equal to orlarger than a predetermined threshold value, it is determined that achange has occurred. Whether image information after encoding or thatbefore encoding is to used by the change detection unit 205 depends onthe difference scheme. Paths of an image input to the reception unit 206are indicated by a solid line and an alternate long and short dashedline in FIG. 2.

Although this embodiment does not depend on the change detection scheme,the inter-frame difference scheme which obtains the absolute value ofthe difference in DCT coefficient in each JPEG coded block is assumedfor the descriptive convenience. In this case, occurrence of changedetection is determined on the basis of three threshold values, i.e.,sensitivity, area ratio, and duration. As for the sensitivity, when thesum of absolute values of DCT coefficient differences in each JPEG codedblock is equal to or larger than the threshold value of sensitivity, itis determined a change has occurred in the block. As the sensitivitybecomes high, the threshold value becomes small, and even a small changeis detected. As for the area ratio, when the ratio of the total area ofblocks in which changes have occurred to the area of the changedetection region defined in an image is not less than a predeterminedthreshold value, it is determined that a changed is detected. As for theduration, when the change detected on the basis of the area ratiocontinues for a predetermined time more than a predetermined thresholdvalue, it is determined that a change is finally detected. Only whenthis determination is done, a change detection event occurs.

As the change amount, the area ratio is selected. The time-series changein area ratio is saved in the change log accumulation unit 207 as achange log. In addition, images obtained when the area ratio is equal toor more than the threshold value and images obtained when the area ratiois smaller than the threshold value are saved in the change logaccumulation unit 207. When the area ratio is equal to or more than thethreshold value, images are accumulated at a predetermined timeinterval. The saved change detection amount and images are transmittedfrom the transmission unit 204 in response to a request from the settingclient.

Examples of the functional blocks of the setting client according tothis embodiment will be described next with reference to FIG. 3. Thesetting client has a live image display function, motion detectionsetting function, motion detection log display function, and accumulatedimage display function. To implement these functions, the setting clientincludes a reception unit 301, display operation unit 302, changedetection setting unit 303, change log display unit 304, andtransmission unit 305. The setting client is implemented by, e.g.,application software mounted on a personal computer. The setting clientdoes not depend on specific hardware or operating system (OS) as long asthe processing capability is sufficient.

An image signal is received by the reception unit 301. The signal isdecoded and displayed by the display operation unit 302. A changedetection result is also received by the reception unit 301 and used bythe change detection setting unit 303 to initialize detection settingdisplay. Then, the set information is displayed on the display operationunit 302. Change log information and accumulated images are received andthen displayed by the change log display unit 304.

The settings to be executed by the display operation unit 302 include,e.g., designation of a change detection region on an image anddesignation of detection sensitivity. The designated values aretransmitted to the change detection setting unit 303 and transmitted tothe camera server through the transmission unit 305. An accumulatedimage request or a change log request is transmitted from the change logdisplay unit 304 to the camera server through the transmission unit 305.

A protocol to acquire a change detection signal or a change detectionsetting signal can be mounted on the TCP protocol or HTTP protocol.However, a detailed description will be omitted because this embodimentdoes not depend on the information acquisition method or protocol.

The GUI (Graphical User Interface) of the setting client will bedescribed next with reference to FIGS. 4 to 6.

FIG. 4 shows an example of a window displayed on the display operationunit 302 of the setting client. It is one window displayed by theapplication software mounted in the window system. The window includes awindow frame 401 and a client region 402. An image received from thecamera server is displayed in the client region 402. A frame 403indicating a change detection target region is displayed in the image.The size of the frame 403 can be changed by dragging latches 404 by amouse cursor (not shown). The position of the frame can also be changedby the mouse cursor (not shown) on the frame. When an “apply” button 406is clicked, the changed value is transmitted to the camera server.

Change detection processing is done by the camera server for the changedetection target region. The position where a change has occurred isrepresented by the position information of a block 405 and transmittedto the client. In this embodiment, as described above, since changedetection is executed for each JPEG block, block display is employed. Ifa change is to be detected for each pixel, the presence/absence of achange is displayed for each pixel. Reference numeral 407 denotes abutton to close the window.

FIG. 5 shows an example of a window displayed by the change detectionsetting unit 303 to execute 15; various kinds of setting for changedetection. Reference numeral 501 denotes an outer frame of the window. Acheck box 502 is used to select whether the change detection function isto be validated or invalidated. An edit box 503 with up and down arrowsis used to designate pan/tilt/zoom for a controllable camera. Referencenumeral 504 denotes sensitivity setting. The value of the sensitivitysetting, which is represented by a number, can be set by moving theslide bar. Reference numeral 507 denotes a threshold value of the arearatio. The value of the area ratio, which is represented by a number,like the sensitivity, can be set by using a slide bar.

As a current area ratio 506, the ratio of the detected region to thedetection target region is indicated in real time by a graph (level bar)which changes to the left or right. With this indicator, a relationbetween the area ratio and the threshold value can be grasped at aglance. An edit box 508 with up and down arrows indicates/sets thethreshold value of duration. A button 509 is used to apply the setvalues and transmit them to the camera server. A button 510 is used toclose the window 501 after the end of setting.

A method of displaying and operating the motion detection log accordingto this embodiment will be described next with reference to FIG. 6. Thewindow shown in FIG. 6 is also displayed on the monitor as one window601. In this window, a log display portion 602, “update” button 607, and“close” button 608 are laid out. In the log display portion 602, a graph604 which time-serially represents the change amount in imagephotographed by the camera along the ordinate is displayed. A thresholdvalue 603 of the change amount, which is used as a criterion of changedetection is indicated by a dotted line. In this embodiment, the arearatio is used as the change amount for the descriptive convenience.However, the change amount is not limited to the area ratio, and anyother amount related to determination of motion detection can be used.The “update” button 607 is used to update the log display. The “close”button 608 is used to close the window 601.

The abscissa of the graph 604 may be indicated by either absolute time(standard time) or relative time. If accumulation is done for a longperiod, it is preferable to switch between full display and partialdisplay or arranged a scaling button. In enlarged display or partialdisplay, the graph is preferably scrolled in the direction of abscissa.In addition, the abscissa of the graph preferably has marks at apredetermined time interval.

When the change amount is not less than the threshold value 603, theimage at that time is saved in the camera server together with the timeinformation. In the setting client, a symbol 605 representing a changedetection image is displayed on the change log graph 604 at a positioncorresponding to the image saving time. When the area ratio is equal toor more than the threshold value, images are accumulated at apredetermined time interval. Hence, the symbols 605 are also displayedin the graph at a predetermined interval at portions where the changeamount is not less than the threshold value 603 on the graph. When thesymbol 605 is designated (clicked) by the pointing device such as amouse, a corresponding saved image 606 is loaded from the camera serverand displayed. The saved image is displayed in a separate window. Theimage only needs to be displayed, and a description of a detaileddisplay form will not particularly be done.

The operation procedures of the camera server will be described nextwith reference to the flowcharts shown in FIGS. 7 and 8. The cameraserver includes a main process for the GUI and a sub process for imageacquisition and change detection. FIG. 7 is a flowchart showing theoperation procedures of the main process of the camera server. FIG. 8 isa flowchart showing the operation procedures of the sub process.

When the main profess shown in FIG. 7 starts, initialization is executedin step S701, and the sub process is activated in step S702. Theoperation procedures of the sub process will be described later withreference to FIG. 8. Then, in step S703, an event is waited. When anevent occurs, processing from step S704 is executed.

In step S704, it is determined whether the event that has occurred is aconnection request event. If YES in step S704, the flow advances to stepS705 to determine whether connection can be permitted. This is becauseif the number of connections is limited, or the connection service timeis limited, connection cannot be permitted. If NO in step S705, the flowadvances to step S706 to execute connection denial notification. Then,the flow returns to step S703 to wait for the next event.

If YES in step S704, the flow advances to step S707 to execute clientregistration processing. In this processing, a management number isassigned to the client, an IP address is registered, and management datasuch as the connection time is initialized. In step S708, a connectionpermission notification is returned to the setting client. For example,the management number assigned to the client and the like are also senttogether. After the end of processing, the flow returns to step S703 towait for the next event.

If NO in step S704, the flow advances to step S709 to determine whetherthe event is a connection end request event. If YES in step S709, theflow advances to step S710 to notify the client of the end ofconnection. After that, the flow advances to step S711 to erase themanagement data of the client. After the end of processing, the flowreturns to step S703 to wait for the next event.

If NO in step S709, the flow advances to step S712 to determine whetherthe event is a change log request event. If YES in step S712, the flowadvances to step S713 to transmit the change log to the client which hasissued the request. Then, the flow returns to step S703 to wait for thenext event.

If NO in step S712, the flow advances to step S714 to determine whetherthe event is an accumulated image request event. If YES in step S714,the flow advances to step S715 to transmit the accumulated image to theclient which has issued the request. Then, the flow returns to step S703to wait for the next event.

If NO in step S714, the flow advances to step S716 to execute otherevent processing. Other event processing includes a live image requestevent, a change detection information request event, a change detectionsetting update request event, and an event of the operating system (OS).A description of this processing will be omitted because it is notdirectly relevant to the present invention. When other processing isended, the flow returns to step S703 to wait for the next event.

The operation procedures of the sub process of the camera server will bedescribed next with reference to FIG. 8. In the sub process, imageacquisition and encoding, motion detection processing, change logaccumulation processing, and image accumulation processing at the timeof motion detection are executed at a predetermined time interval.

When the sub process starts, an image is acquired from the camera instep S801. In step S802, the image is encoded, and the encoded image isstored in an image buffer. If an image request event has occurred in themain process, the image in the image buffer is transmitted. In stepS803, it is determined whether change detection processing is to beexecuted. Whether change detection processing is to be executed can beset by the user by using the “validate change detection” check box 502on the setting window of the client described with reference to FIG. 5.If YES in step S803, processing from step S804 is executed. If NO instep S803, the flow returns to step S801.

In the sub process, two buffers, i.e., detection buffer 1 and detectionbuffer 2 are used for change detection processing. In step S804, data iscopied from detection buffer 1 to detection buffer 2. In step S805, thecurrent image is stored in detection buffer 1. In step S806, changedetection is executed. Several schemes can be used for change detection,as described above. In the example to be described in this embodiment,the ratio of the detection region to the detection target region, i.e.,the area ratio is obtained as the change amount.

The flow advances to step S807 to transmit information such as a changedetection block, area ratio, and detection result to the setting clientunder connection as change detection information. The setting clientreceives and displays these pieces of information. In step S808, thechange log information is saved.

The flow advances to step S809 to determine whether a change isdetected. If YES in step S809, the flow advances to step S810 toaccumulate and save the acquired image as the image at the time ofdetection together with time information obtained from the CPU or OS.The image accumulation/saving method is not particularly limited. Aplurality of frames may continuously or intermittently be saved. If NOin step S809, the flow advances to step S811.

In step S811, it is determined whether the processing is to be ended.Normally, the flow returns to step S801 to continue the processingwithout ending it. If some exceptional processing takes place or an endinstruction is input from the main process, the process is ended.

The operation procedures of the setting client will be described belowwith reference to the flowchart shown in FIG. 9.

First, in step S901, initialization is executed. in step S902, aconnection request is sent to the camera server. In step S903, aresponse from the camera server is waited, and it is determined whetherconnection has successfully be done. If NO in step S903, the flowadvances to step S904 to execute end processing so that the processingis ended. On the other hand, if YES in step S903, the flow advances tostep S905 to wait for an event.

If an event has occurred in step S905, the flow advances to step S906 todetermine whether the event is a log acquisition request event. The logacquisition request event occurs when the setting client is activated,or the “update” button 607 described with reference to FIG. 6 isclicked. If YES in step S906, the flow advances to step S907 to acquirethe change log from the camera server. In step S908, the change log isdisplayed. Then, the flow returns to step S905 to wait for the nextevent.

If NO in step S906, the flow advances to step S909 to determine whetherthe event that has occurred is an accumulated image request event. Theaccumulated image request event occurs when the symbol 605 on the logdisplay graph 604, which corresponds to an image, is selected (clicked)by the user by the pointing device, as described with reference to FIG.6. If YES in step S909, the flow advances to step S910 to acquire theaccumulated image from the camera server. In step S911, the acquiredimage is displayed. Then, the flow returns to step S905 to wait for thenext event.

If NO in step S909, the flow advances to step S912 to determine whetherthe event that has occurred is a scale change request event. The scalechange request event occurs when the window size is changed. If YES instep S912, the flow advances to step S913 to execute scale changeprocessing. In this processing, the vertical or horizontal size of thechange log display graph is changed. When the vertical size is changed,the display magnification in the vertical direction is simply changed.When the horizontal size is changed, the interval of symbolsrepresenting accumulated images also changes. If the interval is toosmall, and the symbols cannot be displayed, they are thinned out.

If NO in step S912, the flow advances to step S914 to determine whetherthe event that has occurred is an end event. If YES in step S914, theflow advances to step S915 to execute end processing. In this endprocessing, a disconnection request is transmitted to the camera server,and the processing is ended.

If NO in step S914, the flow advances to step S916 to execute otherevent processing. Other event processing includes image displayprocessing and motion detection state display processing associated withthe display window shown in FIG. 4, and change detection informationreception processing and change detection setting transmissionprocessing related to the display window shown in FIG. 4 and the settingwindow shown in FIG. 5. A description of this processing will be omittedbecause it is not directly relevant to the motion detection log displayprocessing. When processing in step S916 is ended, the flow returns tostep S905 to wait for the next event.

As is apparent from the above description, according to this embodiment,in an image change detection system including a camera server whichexecutes acquisition of an image photographed by a camera, changedetection processing of the acquired image, image accumulation, andimage transmission, and a setting client which receives the image fromthe camera server through a network and displays the image, and executessetting of the change detection processing, the setting client candisplay the change log graph and images accumulated at the time ofchange detection in association with the graph. Hence, a time-serieschange in photographed image and information about an image when achange in a predetermined amount or more is detected can simultaneouslybe displayed such that they can easily visually be recognized.

Second Embodiment

The second embodiment of the present invention will be described below.In the second embodiment, an image change detection system similar tothat of the first embodiment will be described. A description of thesame parts as in the first embodiment will be omitted. Characteristicparts of the second embodiment will mainly be described.

The second embodiment is different from the first embodiment in themotion detection log display window. FIG. 10 is a view showing themotion detection log display window according to the second embodiment.

In this embodiment, as shown in FIG. 10, in a log display portion 1002which displays a change log in a window 1001 displayed on the monitor ofa setting client, a plurality of kinds of symbols are used in accordancewith the image accumulation timing as symbols representing accumulatedimages at the time of change detection, which are displayed in relationto a graph 1004. In the illustrated example, an image at the start ofchange detection is represented by a symbol 1005. An image during changedetection is represented by a symbol 1006. An image at the end of changedetection is represented by a symbol 1007. The symbols have differentshapes. Accordingly, the start or end timing of change detection canmore visually be expressed. The number of kinds of symbols to be used isnot limited to three. The plurality of kinds of symbols need not havedifferent shapes but have different colors.

When one of the symbols 1005 to 1007 is designated (clicked) by apointing device, a corresponding accumulated image is loaded from thecamera server and displayed in a separate window 1009, as in the firstembodiment. A threshold value 1003, “update” button 1011, and “close”button 1012 are also the same as in the first embodiment.

In this embodiment, the accumulated image at the start of changedetection is displayed in the window 1001 as a thumbnail image 1008together with the graph. The outline of the change log can easily beunderstood without displaying an image with a large size. Even when thethumbnail image 1008 is selected (clicked) by the pointing device, anenlarged image is displayed in the separate window 1009, like thesymbols 1005 to 1007.

The arrangement of the hardware and functional blocks (software) of thisembodiment is the same as in the first embodiment, and a descriptionthereof will be omitted. The operation procedures are partiallydifferent in the setting client. In change log acquisition in step S907and log display in step S908 in the flowchart shown in FIG. 9,simultaneously as a log is acquired, an image at the start of changedetection is acquired, and its thumbnail image is generated anddisplayed. When the thumbnail image is selected by the pointing device,the already acquired image is displayed in its original size.

As is apparent from the above description, according to this embodiment,different symbols are used at the start and end of change detection andduring the change detection as symbols representing accumulated images.In addition, the accumulated image at the start of change is displayedas a thumbnail image. Accordingly, the outline of the change log caneasily visually be understood.

Third Embodiment

The third embodiment of the present invention will be described below.In the third embodiment, an image change detection system similar tothose of the first and second embodiments will be described. Adescription of the same parts as in the above embodiments will beomitted. Characteristic parts of the third embodiment will mainly bedescribed.

In the third embodiment, symbols representing accumulated images at thetime of motion detection are displayed as a set in each period when themotion amount is continuously not less than the threshold value. FIG. 11is a view showing a motion detection log display window according to thethird embodiment.

As shown in FIG. 11, in this embodiment, in a log display portion 1102which displays a change log in a window 1101 displayed on the monitor ofa setting client, symbols representing accumulated images at the time ofchange detection, which are displayed in relation to a graph 1104, aredisplayed in the same color or pattern (design) in a range (also calleda change range) from the start when the change amount is not less thanthe threshold value to the end when the change amount is less than thethreshold value such that the symbols can be regarded as a group. In theillustrated example, symbols 1105 and 1106 in the first change range areexpressed by the first color. Symbols 1107 in the second change rangeare expressed by the second color. Symbols 1108 in the third changerange are expressed by the third color.

To improve readability of the change range, the symbols may be connectedby a line from the start to the end of change, like the symbols 1107.Alternatively, the symbols in the change range may be put in a box, likethe symbols 1108. The colors or patterns of symbols are not limited tospecific colors or patterns. The line that connects the symbols or thebox in which the symbols are put is not limited to a specific shape,either.

When one of the symbols 1105 and 1106 is designated (clicked) by apointing device, a corresponding accumulated image is loaded from thecamera server and displayed in a separate window, as in the firstembodiment. A threshold value 1103, “update” button 1109, and “close”button 1110 are also the same as in the first embodiment.

The arrangement of the hardware and functional blocks (software) of thisembodiment is the same as in the first embodiment, and a descriptionthereof will be omitted. As for the operation procedures, the displayshown in FIG. 11 is done in log display in step S908 in the flowchart ofFIG. 9 which shows the operation procedures of the setting client in thefirst embodiment.

As described above, according to this embodiment, in the change log,symbols are displayed in different colors or patterns for each changerange in which the change amount is not less than the threshold value.Hence, the change range is displayed with better readability.

Fourth Embodiment

The fourth embodiment of the present invention will be described below.In the fourth embodiment, an image change detection system similar tothose of the first to third embodiments will be described. A descriptionof the same parts as in the above embodiments will be omitted.Characteristic parts of the fourth embodiment will mainly be described.

In the fourth embodiment, change logs in a plurality of detection targetregions are displayed on the monitor of a setting client. Morespecifically, in this embodiment, a plurality of frames 403 eachindicating a change detection target region are set on the image in aclient region 402 described with reference to FIG. 4. In the cameraserver, change detection processing is independently executed for eachdetection target region, and the change log and images are accumulatedin accordance with the detection result.

FIG. 12 is a view showing a motion detection log display windowaccording to this embodiment. As shown in FIG. 12, in a log displayportion 1202 which displays a change log in a window 1201 displayed onthe monitor of a setting client, the change log of one of the pluralityof detection target regions is displayed as a graph 1205. Each of theplurality of detection target regions is assigned a number. Thedetection target region to be displayed is set by selecting a numberfrom a list box 1203. In the illustrated example, three detection targetregions are set. The graph of a change log corresponding to the firstdetection target region is displayed. Symbols representing accumulatedimages at the time of change detection are displayed together with thenumbers of the three detection target regions. Referring to FIG. 12,reference numeral 1206 denotes a symbol group of accumulated imagescorresponding to the first detection target region; and 1207, a symbolgroup of accumulated images corresponding to the third detection targetregion.

When one symbol in the symbol groups 1206 and 1207 is designated(clicked) by a pointing device, a corresponding accumulated image isloaded from the camera server and displayed in a separate window, as inthe first embodiment. A threshold value 1204, “update” button 1208, and“close” button 1209 are also the same as in the first embodiment.

The arrangement of the hardware and functional blocks (software) of thisembodiment is the same as in the first embodiment, and a descriptionthereof will be omitted. The operation procedures are as follows. In theoperation procedures of the camera server according to the firstembodiment, the sub process of the camera server shown in FIG. 8 isexecuted for each detection target region. Then, the display shown inFIG. 12 is done in log display in step S908 in the flowchart of thesetting client shown in FIG. 9.

As described above, according to this embodiment, one of the change logsof a plurality of detection target regions can selectively be displayed.In addition, symbols representing accumulated images at the time ofchange detection in the plurality of detection target regions can bedisplayed simultaneously.

Modifications to Embodiments

In the first to fourth embodiments, the present invention is applied toan image change detection system including a camera server and a settingclient. The present invention can also be applied to one device (e.g.,an image change diction apparatus) having a function of detecting achange in photographed image and accumulating a change log and detectedimages.

In the above-described embodiments, a change in image data photographedby one camera is detected, and a change log and detected images areaccumulated. The present invention can also be applied to an arrangementwhich detects a change in each of a plurality of image data photographedby a plurality of cameras and accumulates a change log and detectedimages. In this case, a plurality of cameras may be connected to onecamera server. Alternatively, a plurality of sets of cameras and cameraservers may be prepared.

Fifth Embodiment

FIG. 13 is a view showing the basic arrangement of an image processingsystem according to the fifth embodiment. The image processing apparatusaccording to this embodiment is roughly divided into a video camera2101, display device 2102, and computer 2100. The video camera 2101senses a moving image in a real space. Each frame image is output to thecomputer 2100 as data.

The display device 2102 includes a CRT or liquid crystal screen todisplay a GUI (to be described later) and the like.

The video camera 2101 and display device 2102 are connected to aninterface unit 2114 of the computer 2100. The basic arrangement of thecomputer 2100 will be described below.

A CPU 2110 controls the entire computer 2100 by using programs and datastored in a RAM 2111 and ROM 2112 and also executes processing to bedescribed later.

The RAM 2111 has an area to temporarily store programs and data loadedfrom an external storage device 2115 or storage medium drive device2116, an area to temporarily store each frame image data output from thevideo camera 2101, and a work area to be used by the CPU 2110 to executevarious kinds of processing.

The ROM 2112 stores a boot program and setting data of the apparatus. Anoperation unit 2113 includes a keyboard and mouse so that various kindsof instructions can be input to the CPU 2110. The interface (I/F) 2114connects the video camera 2101 and display device 2102 to the computer2100. The frame image data output from the video camera 2101 is outputto the RAM 2111 through the I/F 2114. An image or character signalcorresponding to image or character data generated on the RAM 2111 bythe computer 2100 is output to the display device 2102 through the I/F2114.

The external storage device 2115 saves programs and data which causesthe OS or CPU 2110 to execute each processing to be described later. Theprograms and data are loaded to the RAM 2111 as needed under control ofthe CPU 2110. Data of moving images sensed in the past by the videocamera 2101 may also be saved and used in processing to be describedlater.

The storage medium drive device 2116 reads out a program or data storedin a storage medium such as a CD-ROM or DVD-ROM and outputs it to theRAM 2111 or external storage device 2115. “Programs and data which causethe CPU 2110 to execute each processing to be described later” may bestored in a storage medium and loaded to the RAM 2111 by the storagemedium drive device 2116.

A bus 2117 connects the above-described components.

The operation of the system having the above arrangement will bedescribed next. In the system according to this embodiment, each frameimage sensed by the video camera 2101 is received, and when thedifference between frame images is equal to or more than a predeterminedvalue, the user is notified of it. In this embodiment, when the numberof pixels whose colors have changed between the fth frame image and the(f+1)th frame image is a predetermined number or more, the user isnotified of it (motion present). In this case, the “predeterminednumber” must be set in advance. Processing for setting the“predetermined number” (to be referred to as a set value hereinafter)will be described below.

Each frame image sensed by the video camera 2101 is input to the RAM2111 through the I/F 2114 and accumulated. Each frame image has a timecode information representing the sensing time, as shown in FIG. 18.

FIG. 18 is a view for explaining time code information. Referring toFIG. 18, reference numeral 2601 denotes time code information attachedto an image. The attachment form is not limited to this.

When a plurality of number of frame images are accumulated in the RAM2111, the CPU 2110 of the computer 2100 generates an average image. Forexample, when 30 frame images are accumulated in the RAM 2111, the pixelvalues at the same position in the frame images are added, and the sumis divided by 30. An image having the product as a pixel value isgenerated as the average image. The image is generally generated byA(i,j)=(F(i,j,l)+F(i,j,2)+ . . . +F(i,j,M))/Mwhere A(i,j) is the pixel value at a position (i,j) in an average imageA (1≦i≦V, 1≦j≦H; V is the horizontal size of the average image A, and His the vertical size of the average image A), and F(i,j,x) is the pixelvalue at the position (i,j) in the xth (1≦x≦M) frame image. Thisequation generates the average image of M frame images. The data of athus generated average image is stored in the RAM 2111.

The CPU 2110 generates an average image every time a predeterminednumber of frame images are accumulated in the RAM 2111. Morespecifically, the frames are divided into groups each including thepredetermined number of frames, and an image is generated by averagingthe frame images in each group. This processing is executed for eachgroup.

A GUI as shown in FIG. 14 is displayed on the display screen of thedisplay device 2102. FIG. 14 is a view showing a display example of aGUI to notify a user that a change equal to or more than a set value (tobe described later) is detected between frame images input from thevideo camera 2101.

Referring to FIG. 14, a window 2201 of the GUI has portions to bedescribed later. In a region 2202, frame images input from the videocamera 2101 are sequentially displayed. The CPU 2110 resizes each frameimage sequentially input from the video camera 2101 to the RAM 2111 anddisplays the frame image in the region 2202.

A message 2203 is displayed when the number of pixels whose colors havechanged between frames is a predetermined number or more. The contentsof the message are not limited to this. If the number of pixels whosecolors have changed less than a predetermined number, a messagerepresenting it may be displayed, or display may be omitted.

GUIs 2204 and 2205 are used to display the set value. The region 2204indicates the possible range of the set value. The slider 2205 indicatesthe position of the set value determined by processing to be describedlater in the region 2204. For example, when the position of the slider2205 in the region 2204 is close to “low” (left end in FIG. 14), theslider 2205 indicates that the set value is large. When the position ofthe slider 2205 in the region 2204 is close to “high” (right end in FIG.14), the slider 2205 indicates that the set value is small.

As described above, the set value indicates the number which the numberof pixels whose colors have changed between frames should exceed todisplay the message “motion present”. When the set value is large, thenotification “motion present” is not displayed unless the number ofpixels whose colors have changed between frames is large. That is, thesensitivity for detecting a change between frame images is low. On theother hand, when the set value is small, the notification “motionpresent” is displayed even when the number of pixels whose colors havechanged between frames is small. That is, the sensitivity for detectinga change between frame images is high.

As described above, the set value determines the sensitivity fordetecting a change between frame images and is used to determine whetherthe notification “motion present” is displayed. This setting isimportant. In this embodiment, the set value is determined by processingto be described later.

A button image 2206 is used to designate the start and end points of asample frame (to be described later) in the CPU 2110. When designationis done by using the keyboard or mouse included in the operation unit2113 (when the button image 2206 is clicked), the CPU 2110 stores, inthe RAM 2111, data representing the time (start time) of detection ofthe instruction. As described above, when a predetermined number offrame images are accumulated in the RAM 2111, the CPU 2110 generates anaverage image. Hence, the CPU 2110 should also have generated theaverage image of a predetermined number of frame images including aframe (start frame) having time code information representing the starttime. As shown in FIG. 17, an average image of M frames (correspondingto the predetermined number of frames) including the start frame will bereferred to as a pre-start average image hereinafter.

When designation of the button image 2206 is stopped (when the buttonimage 2206 is clicked again to release the pressed state), the CPU 2110stores, in the RAM 2111, data representing the time (end time) ofdetection of release of the pressed state of the button image 2206. Asdescribed above, when a predetermined number of frame images areaccumulated in the RAM 2111, the CPU 2110 generates an average image.Hence, the CPU 2110 should also have generated the average image of apredetermined number of frame images including a frame (end frame)having time code information representing the end time. As shown in FIG.17, the average image of M frames (corresponding to the predeterminednumber of frames) including the end frame will be referred to as anafter-end average image hereinafter.

Upon detecting that the pressed state of the button image 2206 isreleased, the computer 2100 starts processing for obtaining the setvalue in accordance with the flowchart shown in FIG. 15.

FIG. 15 is a flowchart of processing for obtaining the set value. Aprogram corresponding to the flowchart shown in FIG. 15 is loaded fromthe external storage device 2115 or from a storage medium to the RAM2111 by the storage medium drive device 2116. When the CPU 2110 executesthe program, the computer 2100 executes processing to be describedlater.

First, frame images each having time code information representing thetime from the start time to the end time are compared with the pre-startaverage image. A frame image (maximum difference image) containingpixels with different pixel values in maximum is specified. The timecode information attached to the maximum difference image is stored inthe RAM 2111. In addition, the number of pixels whose pixel values havechanged between the maximum difference image and the pre-start averageimage is stored in the RAM 2111 (step S2301).

For example, let st be the start time, et be the end time, ft be frameimages (i.e., frame images from the start frame to the end frame) eachhaving time code information representing a time t (st≦t≦et), and sI bethe pre-start average image. The pixel values of corresponding pixelsare compared between the image ft and image sI. If the pixel values aredifferent, a count value C is incremented by one. When this processingis executed for all corresponding pixels, the number (count value C) ofpixels having different pixel values between the image ft and the imagesI can be counted. When this processing is executed for all values tthat satisfy st≦t≦et, the count value C can be obtained for each frameimage from the start frame to the end frame. The frame image with themaximum count value C is the image having the maximum difference fromthe pre-start average image in the frame images from the start frame tothe end frame.

The time code information attached to the maximum difference imagespecified in the above way is stored in the RAM 2111. Simultaneously,the number (maximum count value C) of pixels having different pixelvalues between the maximum difference image and the pre-start averageimage is stored in the RAM 2111.

Next, the frame images each having time code information representingthe time from the start time to the end time are compared with theafter-end average image. A frame image (maximum difference image)containing pixels with different pixel values in maximum is specified.The time code information attached to the maximum difference image isstored in the RAM 2111. In addition, the number of pixels whose pixelvalues have changed between the maximum difference image and theafter-end average image is stored in the RAM 2111 (step S2302). Theprocessing in step S2302 is done by executing the same processing as instep S2301 while “pre-start average image” in the description is changedto “after-end average image”.

It is determined whether the frame specified in step S2301 equals theframe specified in step S2302 (step S2303). This comparison is done bycomparing the time code information specified in the respective stepsand determining whether the pieces of time code information indicate thesame time.

If NO in step S2303, the flow advances from step S2303 to S2304 toselect, of the count values stored in the RAM 2111 in step S2301 or thecount values stored in the RAM 2111 in step S2302, a larger count valueis selected (step S2304). The selected count value C is set as a maximumsensitivity set value (step S2305).

If YES in step S2303, the flow advances from step S2303 to S2305 to setthe data of the count value C stored in the RAM 2111 in step S2301 orS2302 as a maximum sensitivity set value (step S2305).

Next, frame images each having time code information representing thetime from the start time to the end time are compared with the pre-startaverage image. A frame image (minimum difference image) containingpixels with different pixel values in minimum is specified. The timecode information attached to the minimum difference image is stored inthe RAM 2111. In addition, the number of pixels whose pixel values havechanged between the minimum difference image and the pre-start averageimage is stored in the RAM 2111 (step S2306). The processing in stepS2306 is done by executing the same processing as in step S2301 while“maximum” in the description is changed to “minimum”.

The frame images each having time code information representing the timefrom the start time to the end time are compared with the after-endaverage image. A frame image (minimum difference image) containingpixels with different pixel values in minimum is specified. The timecode information attached to the minimum difference image is stored inthe RAM 2111. In addition, the number of pixels whose pixel values havechanged between the minimum difference image and the after-end averageimage is stored in the RAM 2111 (step S2307). The processing in stepS2307 is done by executing the same processing as in step S2306 while“pre-start average image” in the description is changed to “after-endaverage image”.

When the difference between two images compared in step S2306 or S2307is too small (for example, when the image size is 320×400, and thedifference is 100 pixels), the result of comparison processing isinvalidated, and the next comparison processing is started.

It is determined whether the frame specified in step S2306 equals theframe specified in step S2307 (step S2308). This comparison is done bycomparing the pieces of time code information specified in therespective steps and determining whether the pieces of time codeinformation indicate the same time.

If NO in step S2308, the flow advances from step S2308 to S2309 toselect, of the count values stored in the RAM 2111 in step S2306 or thecount values stored in the RAM 2111 in step S2307, a smaller count valueis selected (step S2309). The selected count value C is set as a minimumsensitivity set value (step S2310).

If YES in step S2308, the flow advances from step S2308 to S2310 to setthe data of the count value C stored in the RAM 2111 in step S2306 orS2307 as a minimum sensitivity set value (step S2310).

The final sensitivity set value as the set value” is obtained by usingthe maximum and minimum sensitivity set values obtained by the aboveprocessing (step S2311). Various methods are available to obtain thefinal sensitivity set value. For example, the average value of themaximum and minimum sensitivity set values is calculated, and theresultant average value is set as the final sensitivity set value. Themaximum sensitivity set value may directly be set as the set value. Theminimum sensitivity set value may directly be set as the set value.

The CPU 2110 moves the slider 2205 in the region 2204 to a positioncorresponding to the final sensitivity set value obtained by the aboveprocessing.

With the set value obtained by the above processing, when a change morethan an average change (the average number of pixels whose pixel valueschange) has occurred between frames, the user is notified of it.

For example, a person passes by the video camera 2101, and its videoimage is input to the computer 2100 through the video camera 2101 as amoving image. The set value obtained by the above processing can beapplied to determine whether a change has occurred in a moving imagesensed by a surveillance camera in a building at night. The change inreal space to be detected is played back in front of the video camera2101, the set value is automatically changed.

As described above, according to this embodiment, the set value can beobtained without manual operation.

In addition, when each of the maximum and minimum sensitivity set valuesis used as a criterion, setting to detect a change equal to or more thana predetermined value and setting not to detect a change less than thepredetermined value can simultaneously be executed.

In this embodiment, the notification “motion present” is “displayed” onthe GUI shown in FIG. 2. However, the present invention is not limitedto this. A speaker may be connected to the computer 2100, and voice dataof the notification message “motion present” may be loaded in the RAM2111 so that voice is output from the speaker in accordance with thevoice data for notification. The notification form is not particularlylimited.

In this embodiment, time code information is used to identify eachframe. Each frame may be identified by another method.

In this embodiment, processing is executed for entire frame image. Insome cases, even a change in part of an image should be detected. Inthis case, the above-described processing is executed for only a knownregion in an image. Accordingly, a set value to do notification when achange has occurred in that region can be obtained.

Sixth Embodiment

To obtain the maximum and minimum sensitivity set values, the averageimage of the pre-start average image and after-end average image isobtained. The maximum and minimum values of the difference (the numberof pixels with different pixel values) between the average image andeach frame from the start frame to the end frame are obtained as themaximum and minimum sensitivity set values.

Seventh Embodiment

In the fifth embodiment, a moving image input from the video camera 2101is processed. However, the present invention is not limited to this.Moving image data loaded from an external storage device 2115 or storagemedium drive device 2116 to a RAM 2111 may be processed.

In this case, instead of sequentially displaying each frame image in aregion 2202, some or all frames may be displayed as a list, as shown inFIG. 16. FIG. 16 is a view showing a display example of a GUI whichdisplays a list of frame images and executes the operation of setting aset value.

In this case, the start and end frames can be designated by designatingthem using the keyboard or mouse as an operation unit 2113. Of the twodesignated positions, the left position is the position of the startframe, and the right position is the position of the end frame.

In this case, frames to be used to obtain the pre-start average imageand after-end average image can also be designated by designating themon the GUI shown in FIG. 16.

In the fifth embodiment, the sensitivity (set value) is determined by asingle parameter. However, even in a motion detection scheme whichdesignates the sensitivity by a plurality of parameters, the sameprocessing as in the fifth embodiment can be executed for one parameter.

For example, an image is segmented into small rectangles. A change inpixel value is detected in each rectangle, and simultaneously, a changebetween the rectangles is detected. In this case, the sensitivity is setas two parameters “intra-rectangle sensitivity” and “inter-rectanglesensitivity”. In this case, one of the “intra-rectangle sensitivity” and“inter-rectangle sensitivity” is fixed or designated by the user. Theother parameter is automatically set by the same method as in the fifthembodiment.

Alternatively, when the “intra-rectangle sensitivity” and“inter-rectangle sensitivity” are internally mapped to one parameter,automatic setting can be executed. In this case, for example, “virtualsensitivity” is introduced, which is given by virtualsensitivity=f(intra-rectangle sensitivity, inter-rectangle sensitivity)(“Virtual sensitivity” is “virtual” because it is not presented to theuser, and “f” is an appropriate function. The pairs of virtualsensitivity and intra-rectangle sensitivity and inter-rectanglesensitivity have a one-to-one correspondence. For example, when theintra-rectangle sensitivity takes a value from 0 to 100, virtualsensitivity=“inter-rectangle sensitivity”×100+“intra-rectanglesensitivity” can be used).

Accordingly, the number of parameters is substantially one. Hence,sensitivity setting can be implemented by the same method as in thefifth embodiment.

In the fifth embodiment, “processing for obtaining the set value” and“processing of executing notification “motion present” in accordancewith the obtained set value” are done by one computer. Instead, theprocessing operations may be distributed to a plurality of computers.

Eighth Embodiment

In the eighth embodiment of the present invention, an example of motiondetection setting will be described, in which in sensitivity setting formotion detection processing by the difference between images, a time inwhich a motion is present and a time in which no motion is present aredesignated for a live image, thereby automatically calculating anddisplaying an appropriate sensitivity range.

Hardware in this embodiment includes a camera and a connectablecomputer. Examples of the video output scheme of the camera are NTSC(National Television System Committee), PAL (Phase Alternation by Linecolor television), and an independent scheme by USB (Universal SerialBus) or IEEE1394 connection. The present invention does not depend onthe video output scheme. The computer has a CPU, RAM, ROM, secondarystorage device, monitor, keyboard, mouse, and external input/outputinterface. The computer only needs to have a function of inputting videodata from the camera. The form of the computer is not particularlylimited, and any commercially available computer, dedicated set-top box,portable information terminal, or cellular phone terminal can be used.

In the following description, the camera and computer are directlyconnected, and setting is done on the computer. Instead, setting may bedone from another computer via a network. Alternatively, the camera mayhave a network connection function so that the computer can acquirevideo data from the camera through a network and execute motiondetection processing.

Software in this embodiment includes an image processing process foracquiring an image from the camera and executing motion detectionprocessing on the computer and a GUI process for displaying the imageand motion detection processing result and executing setting. The GUIprocess is the main process, and the image processing process is the subprocess. A process is a program execution unit. A process may includeeither a plurality of programs or an execution unit called a thread in asingle program.

The outline of motion detection processing by a difference will bedescribed next with reference to FIG. 19. An image sensing unit 3011 isa camera which inputs video data. An image acquired from the camera isused for reference image generation by a reference image generation unit3012. The reference image changes depending on the difference algorithm.Typical examples of difference processing are inter-frame difference orbackground difference. In inter-frame difference, the reference image isan image one or a plurality of frames before. In background difference,for example, the average image or median image of a plurality of framesduring a past time period without any motion is used.

A difference processing unit 3013 executes difference processing for theinput image and reference image to calculate a change amount. In typicaldifference processing, the sum of the absolute differences between theinput image and the reference image is calculated as the change amount.For example, the RGB components or YCbCr components of pixels or thecode amounts or DCT coefficients of JPEG coded blocks are used for thedifference operation.

A discrimination processing unit 3014 discriminates whether the obtaineddifference amount corresponds to a state “motion present”. In mostcases, when the difference amount is not less than a predeterminedthreshold value, the state is regarded as “motion present”. If thedifference amount is equal to or less than the threshold value, thestate is regarded as “no motion”. The threshold value can be set orchanged by the user by a setting unit 3015. The threshold value is oftenconverted into “sensitivity” by the setting unit 3015, and displayed ordesignated. When the threshold value is large, the sensitivity is low.When the threshold value is small, the sensitivity is high. For example,the relationship between sensitivity S and a threshold value Th is givenbyS=a(Thmax−Th)+bwhere Thmax is the maximum value of the threshold value, and a and b areconstants.

An example of the motion detection scheme by typical difference has beendescribed above. The method of this embodiment executes differenceprocessing on the basis of a change amount and a threshold value anddoes not depend on the type of difference operation of obtaining achange amount.

An algorithm for obtaining an appropriate threshold value range bydesignating a change amount and the presence/absence of motion in movingimages will be described next with reference to FIGS. 20A and 20B.

FIG. 20A is a graph showing a change in change amount over time. Theabscissa represents time, and the ordinate represents a change amount.Times from T1 to T2 and from T5 to T6 without motion and time from T3 toT4 with motion are designated. The change amount largely changes betweena state with motion and that without motion. Hence, a lower limit Th1 ofan appropriate threshold value can be determined from the maximum valueof the change amount during the period without motion, and an upperlimit Th2 of the appropriate threshold value can be determined from theminimum value of the change amount during the period with motion. As aresult, the user sets the appropriate threshold value Th between thelower limit Th1 and the upper limit Th2 of the threshold value.

The lower limit Th1 and upper limit Th2 of the appropriate thresholdvalue can also be set by another method. FIG. 20B shows the detectionerror ratio and detection miss ratio when the threshold value ischanged. A detection error means that a state without motion iserroneously discriminated as a state with motion. A detection errorratio curve 3021 is generated from the distribution of change amountsduring the no motion detection times in FIG. 20A. When the thresholdvalue is large, the detection error ratio is low. Detection miss meansthat a state with motion is erroneously as a state without motion. Adetection miss ratio curve 3022 is generated from the distribution ofchange amounts during the motion detection times in FIG. 20A. When thethreshold value is small, the detection miss ratio is low.

The sum of the detection error ratio and detection miss ratio indicatesthe total accuracy of detection, which is indicated by a detection ratiocurve 3023 in FIG. 20B. As a consequence, the threshold value range tokeep a predetermined discrimination accuracy E is the range from Th1 toTh2. According to this method, discrimination is possible even when thechange amount distribution when “no motion” is designated and that when“motion present” is designated overlap.

The GUI for motion detection sensitivity setting will be described nextwith reference to FIGS. 21A and 21B. FIG. 21A is a view showing anexample of a dialog window for sensitivity setting. The dialog includesa preset selection portion 3031, image display portion 3032, detectiontarget region setting frame 3033, “motion present” button 3034, “nomotion” button 3035, change amount/discrimination result display portion3036, sensitivity adjusting portion 3037, “OK” button 3038, and “cancel”button 3039.

The preset selection portion 3031 selects preset that designates pan,tilt, and zoom of a controllable camera. This designation is unnecessaryfor a stationary camera, as a matter of course. The motion detectiontarget region 3033 is a frame which designates a region in an image tobe subjected to difference processing. The position and size of thisregion can be changed by dragging the mouse cursor.

The “motion present” button 3034 and “no motion” button 3035 designatethe start and end times of “motion present” state and “no motion” state.The buttons are set in a pressed state by one click and returned to thenormal state by the second click. The start time is designated byclicking the button. The end time is designated by releasing the pressedstate. After the designation, the data of “motion present” and “nomotion” times are saved in the RAM or secondary storage device. When thepressed state of the “motion present” button is released to designatethe end of the “motion present” state, the upper limit of theappropriate threshold value is calculated. When the pressed state of the“no motion” button is released to designate the end of the “no motion”state, the lower limit of the appropriate threshold value is set.

The change amount/discrimination result display portion 3036 displayschange amounts as a graph and also displays the discrimination result ofthe presence/absence of motion by colors. The sensitivity adjustingportion 3037 converts the threshold value into sensitivity. Thesensitivity can be set by dragging the knob of the slider. On theslider, the value range from minimum appropriate sensitivity S1 tomaximum appropriate sensitivity S2 is highlighted, as shown in FIG. 21B.As already described above, the minimum value S1 of the appropriatesensitivity is obtained from the maximum value Th2 of the appropriatethreshold value. The maximum value S2 of the appropriate sensitivity isobtained from the minimum value Th1 of the appropriate threshold value.

The operation procedures of the motion detection setting program will bedescribed next with reference to FIGS. 22A, 22B, and 23. FIGS. 22A and22B show the operation procedures of the main process including the GUI.FIG. 23 shows the operation procedures of the sub process for executingmotion detection processing.

In the main process shown in FIGS. 22A and 22B, after the start,initialization is executed in step S3401. In step S3402, the sub processis activated. When initialization is ended, a dialog as shown in FIG.21A is displayed. In step S3403, an event is waited. Only major eventsrelevant to the present invention will be described below.

When an event occurs, it is determined in step S3404 whether the eventis preset input processing. This event occurs when the preset selectionportion 3031 is changed. If YES in step S3404, the flow branches to Y.In step S3405, preset setting is executed. In step S3406, camera controlis executed. Then, the flow returns to step S3403 to wait for the nextevent.

If NO in step S3404, the flow branches to N. In step S3407, it isdetermined whether the event is a region change event. This event occurswhen the motion detection target region frame 3033 is changed. If YES instep S3407, the flow branches to Y. The flow advances to step S3408 toexecute region change processing. Then, the flow returns to step S3403to wait for the next event.

If NO in step S3407, the flow branches to N. In step S3409, it isdetermined whether the event is a “motion present” button click event.If YES in step S3409, the flow branches to Y. In step S3410, the “motionpresent” start time is stored, and the flow returns to step S3403. If NOin step S3409, the flow branches to N. In step S3411, it is determinedwhether the event is a “motion present” button pressed state releaseevent. If YES in step S3411, the flow branches to Y. The flow advancesto step S3412 to store the “motion present” end time. In step S3413, theupper limit value of the appropriate threshold value is calculated. Instep S3414, sensitivity setting display is updated. The flow returns tostep S3403 to wait for the next event.

If NO in step S3411, the flow branches to N. The flow advances to stepS3415 to determine whether the event is a “no motion” button clickevent. If YES in step S3415, the flow branches to Y. The flow advancesto step S3416 to store the “no motion” start time, and the flow returnsto step S3403. If NO in step S3415, the flow branches to N. The flowadvances to step S3417 to determine whether the event is a “no motion”button pressed state release event. If YES in step S3417, the flowbranches to Y. The flow advances to step S3418 to store the “no motion”end time. In step S3419, the lower limit value of the appropriatethreshold value is calculated. Then, processing from step S3414 isexecuted.

If NO in step S3417, the flow advances to step S3420 to determinewhether the event is a detection setting update request. This eventoccurs when the “OK” button 3038 is clicked. If YES in step S3420, theflow branches to Y. The flow advances to step S3421 to update the motiondetection set value. Otherwise, the flow returns to step S3403 to waitfor the next event.

The operation principle of the sub process for image processing will bedescribed next with reference to FIG. 23. After the start, an image isacquired in step S3501. The image is input from the camera or acquiredthrough the network. In step S3502, it is determined whether referenceimage necessary for difference processing is present.

As the reference image, at least one image in the past is used, asdescribed above. If NO in step S3502, the flow branches to N. The flowadvances to step S3506 to generate and update the reference image. Forbackground difference, a “no motion” state is necessary for generating areference image. This can be realized by adjusting the timing ofsetting. The timing can also be adjusted by adding, e.g., a “startsetting” button to the setting dialog. This is not relevant to thepresent invention, and a description thereof will be omitted.

If YES in step S3502, the flow branches to Y. The flow advances to stepS3503 to execute difference processing. In step S3504, motiondiscrimination processing is executed. In step S3505, change amountdisplay update processing is executed. In step S3506, the referenceimage is updated. In step S3507, the window display is updated. Indisplaying the image, a region with a change may be highlighted.Finally, if no end instruction is input in step S3508, the flow branchesto N. The flow returns to step S3501 to repeat the processing.

Program processing also includes processing executed when the “cancel”button 3039 is clicked. When setting is canceled, the set value is notdetermined, and the dialog is ended. A description of processing forevents unique to the operating system will be omitted.

As is apparent from the above description, according to this embodiment,in motion detection based on the difference between images, the userdesignates the start and end times of a “motion present” state and thoseof a “no motion” state while observing images, thereby obtaining anappropriate threshold value range. In addition, since the upper andlower limits of the appropriate sensitivity value are displayed, theuser can more easily set the threshold value and sensitivity.

Ninth Embodiment

As the ninth embodiment of the present invention, a method of setting anappropriate threshold value and sensitivity by inputting the times of“motion present” and “no motion” states of accumulated video images, asin the eighth embodiment, will be described. Since an over-time shiftgraph of change amounts is displayed together with the accumulated videoimages, the “motion present” and “no motion” periods can moreconveniently be set.

The hardware and software configurations of this embodiment are the sameas in the eighth embodiment, and a description thereof will be omitted.FIG. 24 shows an example of a GUI for sensitivity setting of thisembodiment. FIG. 24 shows a dialog window which includes an imagedisplay portion 3601, motion detection target region setting portion3602, accumulated video selection portion 3616, “motion present” button3603, “no motion” button 3604, change amount/motion discriminationresult display portion 3605, sensitivity adjusting portion 3606,representative image display portions 3607, 3608, and 3609, changeamount shift graph 3610, motion time indication bar 3612, timeindication bar 3611, playback control button group 3613, “OK” button3614, and “cancel” button 3615. To store an accumulated video image on alive image window, a new button must be added independently of FIG. 24,though a description thereof will be omitted.

An operation method will be described next. First, an arbitrary videoimage is selected by the accumulated video selection portion 3616. Thevideo image is displayed in the image display portion 3601, and playbackstarts. During playback, a change amount is displayed in the changeamount display portion 3605. The position and size of the motiondetection target region 3602 are changed by dragging the mouse, therebydesignating the motion detection target region. When the playback buttonis clicked again during playback, playback is stopped.

When the “motion present” button 3603 and “no motion” button 3604 areoperated at appropriate timings during playback, the start and end timesof “motion present” and “no motion” states are designated, as in theeighth embodiment. In this embodiment, however, when the “motionpresent” button 3603 is clicked, display of the motion time indicationbar 3612 is started in the change amount shift graph 3610. When thepressed state is released, the length of the bar is determined. The“motion present” state and the “no motion” state can be distinguished bythe colors or patterns of the bar. The start and end times are alsodisplayed on the time indication bar 3611. The above processing alsoapplies to click of the “no motion” button 3604 and release of itspressed state.

When the motion time indication bar 3612 is determined, representativeimages during the period are displayed like 3607, 3608, and 3609 incorrespondence with the indication of the bar. In detecting the “motionpresent” state, an image at a time with the maximum change amount can beused. Alternatively, images at the start of the “motion present” and “nomotion” states may simply be used. In this embodiment, therepresentative images 3607, 3608, and 3609 corresponding to all the “nomotion” and “motion present” periods are displayed. Instead, one of themmay be displayed. The start and end times can be changed by dragging themotion period indication bar 3612 in the horizontal direction. In thisembodiment, the motion period indication bars 3612 corresponding to allthe “no motion” and “motion present” periods are displayed. Instead, oneof them may be displayed. When the start and end times are changed,display of the sensitivity display portion 3606 and representative imagedisplay portions 3607 to 3609 is also updated.

The playback time can be changed by dragging the knob of the timeindication bar 3611. With the playback control button group 3613,playback, fast-forward, and rewinding operations can be performed. Theseoperations are the same as in general moving image playback software.

When the start and end times of the “motion present” and “no motion”states are determined, the lower and upper limits of the appropriatethreshold value are determined and reflected on the highlighted portionof the sensitivity setting slider 3606. The algorithm for determiningthe appropriate threshold value range is the same as in the eighthembodiment. The operations of the change amount/discrimination resultdisplay portion 3605, sensitivity setting slider 3606, “OK” button 3614,and “cancel” button 3615 are also the same as in the eighth embodiment,and a description thereof will be omitted.

The operation procedures of the motion detection sensitivity settingprogram according to this embodiment will be described next withreference to FIGS. 25 and 26. FIG. 25 shows the operation procedures ofthe main process. The process composition of this embodiment is the sameas that of the eighth embodiment, and the operation is also the same inmany parts. Points different from the eighth embodiment will mainly bedescribed below.

After the start, initialization is executed in step S3701. In stepS3702, the image processing process is activated. In step S3703, anevent is waited. When an event occurs, it is determined in step S3704whether the event is a playback control event. This event occurs whenthe playback, fast-forward, or rewinding button 3613 or the timedesignation bar 3611 is operated. If YES in step S3704, the flowbranches to Y. In step S3705, playback control processing is executed.Motion detection processing is not executed during fast-forward orrewinding.

If NO in step S3704, the flow branches to N. The flow advances to stepS3706 to determine whether the event is a detection region change input.If YES in step S3706, the flow branches to Y. In step S3707, theposition and size of the changed region are stored as detection regionchange processing.

If NO in step S3706, the flow branches to N. In step S3708, it isdetermined whether the event is an event related to the “motion present”or “no motion” button. The event processing (step S3708) and operation(step S3709) when the event has occurred are the same as the operationprocedures of the first embodiment (steps S3409 to S3419) and thereforeare not illustrated. As described above, when the start and end timesare determined, display of the time indication bar 3611, motion timeindication bar 3612, and representative image display portions 3607 to3609 is updated. In addition, the appropriate threshold value rangechanges, and display of the sensitivity setting slider 3606 alsochanges.

If NO in step S3708, the flow branches to N. In step S3710, it isdetermined whether the event is motion time indication bar change event.This event occurs when the position and size of the motion timeindication bar 3612 are changed by dragging it by the mouse. If YES instep S3710, motion time indication bar change processing is executed instep S3711. The position and size are converted into start and endtimes. When the start and end times are changed, display of the motionperiod indication bar 3612 and representative image display portions3607 to 3609 is also changed in synchronism. In addition, theappropriate threshold value range changes, and display of thesensitivity setting slider 3606 also changes.

If NO in step S3710, the flow branches to N. The flow advances to stepS3712 to determine whether the event is a motion detection settingchange event. This event is the same as the processing in step S3420 inFIG. 22B of the eighth embodiment and occurs when the “OK” button 3614is clicked. If YES in step S3712, the flow branches to Y. In step S3713,the motion detection set value is updated.

FIG. 26 shows the operation procedures of the sub process for imageprocessing. The operation procedures include the same processing as inthe sub process of the eighth embodiment. Points different from theeighth embodiment will mainly be described. After the start, it isdetermined in step S3801 whether a stop state is set. If YES in stepS3801, the flow branches to Y. The flow advances to step S3810 todetermine whether an end instruction event has occurred. If YES in stepS3810, the flow branches to Y, and the processing is ended. Otherwise,the flow branches to N. The flow returns to step S3801 to continue theprocessing.

If NO in step S3801, the flow branches to N. It is determined in stepS3802 whether a playback state is set. If YES in step S3802, the flowbranches to Y. It is determined in step S3803 whether a frame remains.If YES in step S3803, the flow branches to Y to execute image differenceprocessing from step S3804. Processing in steps S3804 to S3810 is thesame as that in steps S3502 to S3508 in FIG. 23 of the eighthembodiment, and a description thereof will be omitted. When theprocessing is ended, the flow returns to step S3801.

If NO in step S3802, a fast-forward or rewinding state is set. Hence,the flow branches to N. The flow advances to step S3812 to executefast-forward or rewinding processing. In step S3809, display is updated.The flow advances to step S3810 to execute end determination processing.If NO in step S3803, the flow advances to step S3811 to change the stateto the stop state. Then, processing from step S3809 is executed.

According to this embodiment, the user inputs a period with motion and aperiod without motion while observing the playback images of accumulatedimages so that the ranges of the appropriate threshold value andsensitivity are automatically calculated. In addition, since theover-time shift of the change amount is displayed as a graph, and thestart and end times of the “motion present” and “no motion” states aregraphically displayed, the “motion present” and “no motion” periods caneasily be set.

10th Embodiment

As the 10th embodiment of the present invention, a method ofautomatically setting the motion detection target region on an image onthe basis of the change amount distribution during the “motion present”period in the eighth embodiment will be described.

The automatic motion detection target region setting method according tothis embodiment will be described with reference to FIG. 27. Referringto FIG. 27, the sum of change amounts during the “motion present” periodis displayed for each partial region on an image 3901. A region withlarge motion is displayed in a dark color. This is a histogram in thedirection of time. When a partial region having a sum equal to or morethan a predetermined value is extracted, and a circumscribed rectangle3902 is obtained, a motion detection target region can be set.

For the GUI of motion detection setting in this embodiment, a check box3108 to instruct automatic setting of the motion detection region isadded to the setting GUI (FIG. 21) of the eighth embodiment. Thedetection target region is determined by the above-described algorithmat a timing when the pressed state of the “motion present” button isreleased.

The operation procedures of the program of this embodiment are the sameas in the eighth embodiment except the processing for obtaining thedetection target region by the above-described method is added to the“motion present” state end processing in step S3412 of the processingprocedures of the main process, and a description thereof will beomitted.

In the above description, the processing is added to the eighthembodiment. Instead, the function of automatically setting the detectiontarget region may be added to the ninth embodiment by the same method asdescribed above.

According to this embodiment, in motion detection based on thedifference between images, the user designates the “motion present”period. The detection target region can automatically be set from thestatistics of the change amounts during the “motion present” period.Accordingly, the motion detection target region setting processing canbe simplified.

As described above, according to this embodiment, in the motiondetection method of detecting motion in an image on the basis of achange amount obtained from the difference between images, the userdesignates the “motion present” and “no motion” periods so that motiondetection setting to calculate an appropriate threshold value andpresent an appropriate sensitivity range to the user can be executed. Anappropriate sensitivity range can be presented not only for a live imagebut also for an accumulated image. In addition, a motion detectionsetting method of designating the “motion present” period andautomatically designating the motion detection target region from thestatistics of change amounts during the period is provided.

11th Embodiment

As the 11th embodiment of the present invention, a method will bedescribed, in which when the start of automatic sensitivity setting isinstructed when an image has no motion, optimum sensitivity isautomatically set from the statistics of difference values for severalsec.

The hardware and software configurations of this embodiment are the sameas in the eighth embodiment, and a description thereof will be omitted.FIG. 29 shows an example of a sensitivity setting window according tothis embodiment. The 11th embodiment is different from the eighthembodiment in that the “motion present” button 3034 and “no motion”button 3035 are omitted, and an automatic setting button 3114 isarranged instead. In addition, since sensitivity is automatically set inthis embodiment, a sensitivity setting slider 3117 indicates no upperand lower limits of sensitivity, unlike the sensitivity setting slider3037. When the automatic setting button 3114 is clicked in a “no motion”state, automatic sensitivity setting starts.

The procedures of automatic sensitivity setting will be described withreference to FIG. 30. FIG. 30 shows the operation procedures related toautomatic sensitivity setting of a motion detection setting program. Theuser confirms that no motion is present on the screen and clicks theautomatic setting button 3114. A “no motion” state is a state in whichno motion to be detected is present. Motion other than a detectiontarget, such as swaying trees or waving water surface, poses no problem.

After the start of motion detection setting, an attempt to acquire animage is made in step S3201. In this embodiment, assume that thereference image described in the eighth embodiment has already beenacquired. If NO in step S3202, the flow branches to N. The flow advancesto step S3203 to determine whether the attempt has continuously failed.The number of continuous failures has an upper limit. If the number isequal to or smaller than the upper limit, the flow branches to N. Theflow returns to step S3201 to execute image acquisition again. If thenumber of continuous failures exceeds the upper limit, the flow branchesto Y. The flow advances to step S3204 to display a failure message, andthe automatic setting is ended. In this case, the sensitivity valueremains the value before the start of automatic setting.

If YES in step S3202, the flow branches to Y. In step S3205, differenceprocessing is executed. As described in the eighth embodiment, thisembodiment does not depend on difference processing. For example, incontinuous frames, the sum of difference for DCT coefficients afterquantization is obtained for a JPEG block having 8×8 pixels of adetection region 3113 and used as the difference value. Continuousframes mean, e.g., a current frame and an immediately preceding frame.

Next, in step S3206, an average difference value Dt (t=1, . . . , n) inthe acquired frames is calculated. Since exceptional values are excludedin obtaining the average, about 10% values on the upper and lower sidesof the difference values are excluded. In step S3207, it is determinedwhether the minimum acquisition time has elapsed. The minimumacquisition time is fixed to, e.g., 2 sec. If NO in step S3207, the flowbranches to N. The flow returns to step S3201 to continue imageacquisition and difference processing.

If YES in step S3207, the flow branches to Y. The flow advances to stepS3208 to determine whether the number of acquired frames is equal to ormore than the minimum number of frames. This processing is done tocalculate the final average difference value by using the averagedifference value of, e.g., at least five frames to obtain astatistically significant value.

If NO in step S3208, the flow branches to N. It is determined in stepS3209 whether the maximum acquisition time has elapsed. The maximumacquisition time is set to, e.g., 10 sec. If YES in step S3209, the flowbranches to Y. In step S2304, a failure message is displayed, and theprocessing is ended. This situation occurs when, e.g., five framescannot be obtained even after 10-sec wait. If NO in step S3209, the flowbranches to N. The flow returns to step S3201 to continue imageacquisition and difference processing.

If YES in step S3208, the flow branches to Y. In step S3210, averagedifference value calculation for all frames is executed. At this time,an average D of the average difference values Dt of the frames isobtained. In this case as well, for example, when 10% values on theupper and lower sides are excluded, noise can be removed. In step S3211,sensitivity is obtained from the difference value. At this time,sensitivity S which is inversely proportional to the average differencevalue D is obtained by using the same equation as in the eighthembodiment.S=a(Dmax−D)+bwhere S: sensitivity, Dmax: maximum difference value, D: averagedifference value, and a, b: constants

In this equation, the constant b acts to reduce the sensitivity andnormally takes a negative value. After sensitivity calculation, amessage representing that automatic sensitivity setting has successfullybe done is displayed in step S3212. In step S3213, the sensitivity isupdated to the new value, and the processing is ended.

This scheme can be applied not only to a stationary camera but also to acontrollable camera capable of pan, tilt, and zoom control. However, ifcamera control which influences images is executed in such a cameraduring automatic sensitivity setting, automatic sensitivity settingfails. Such camera control includes, e.g., pan, tilt, zoom, focus, gain,exposure, aperture, shutter speed, and white balance.

In this embodiment, only the start of automatic sensitivity setting isdesignated. This assumes that no motion is present in the screen duringautomatic sensitivity setting. For this reason, if unexpected largemotion continuously occurs in the screen, this motion cannot always beremoved as noise. Hence, it may be impossible to set a correct value.However, when a stable “no motion” is set for several sec, sensitivitycan automatically be obtained from the difference amount during thattime.

Other Embodiments

In each of embodiments described above, for change and motion detection,whether the change amount of image data is less than a predeterminedvalue is detected. However, the present invention is not limited in thisway. For example, the detection can be based on whether the changeamount of image data exceeds a predetermined value, and when the changeamount exceeds the predetermined value, then the next processing andcontrol (notification processing, storing the image data, and etc) canbe executed.

Furthermore, the invention can be implemented by supplying a softwareprogram, which implements the functions of the foregoing embodiments,directly or indirectly to a system or apparatus, reading the suppliedprogram code with a computer of the system or apparatus, and thenexecuting the program code. In this case, so long as the system orapparatus has the functions of the program, the mode of implementationneed not rely upon a program.

Accordingly, since the functions of the present invention areimplemented by computer, the program code installed in the computer alsoimplements the present invention. In other words, the claims of thepresent invention also cover a computer program for the purpose ofimplementing the functions of the present invention.

In this case, so long as the system or apparatus has the functions ofthe program, the program may be executed in any form, such as an objectcode, a program executed by an interpreter, or scrip data supplied to anoperating system.

Example of storage media that can be used for supplying the program area floppy disk, a hard disk, an optical disk, a magneto-optical disk, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, a non-volatile type memorycard, a ROM, and a DVD (DVD-ROM and a DVD-R).

As for the method of supplying the program, a client computer can beconnected to a website on the Internet using a browser of the clientcomputer, and the computer program of the present invention or anautomatically-installable compressed file of the program can bedownloaded to a storage medium such as a hard disk. Further, the programof the present invention can be supplied by dividing the program codeconstituting the program into a plurality of files and downloading thefiles from different websites. In other words, a WWW (World Wide Web)server that downloads, to multiple users, the program files thatimplement the functions of the present invention by computer is alsocovered by the claims of the present invention.

It is also possible to encrypt and store the program of the presentinvention on a storage medium such as a CD-ROM, distribute the storagemedium to users, allow users who meet certain requirements to downloaddecryption key information from a website via the Internet, and allowthese users to decrypt the encrypted program by using the keyinformation, whereby the program is installed in the user computer.

Besides the cases where the aforementioned functions according to theembodiments are implemented by executing the read program by computer,an operating system or the like running on the computer may perform allor a part of the actual processing so that the functions of theforegoing embodiments can be implemented by this processing.

Furthermore, after the program read from the storage medium is writtento a function expansion board inserted into the computer or to a memoryprovided in a function expansion unit connected to the computer, a CPUor the like mounted on the function expansion board or functionexpansion unit performs all or a part of the actual processing so thatthe functions of the foregoing embodiments can be implemented by thisprocessing.

If the present invention is realized as a storage medium, program codescorresponding to at least one of the above mentioned flowcharts (FIGS.7-9, 15, 22A, 22B, 23, 25, 26 and FIG. 30) is to be stored in thestorage medium.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the claims.

CLAIM OF PRIORITY

This application claims priority from Japanese Patent Applications No.2003-311340 filed on Sep. 3, 2003, No. 2003-371039 filed on Oct. 30,2003, No. 2003-415428 filed on Dec. 12, 2003, and No. 2004-141239 filedon May 11, 2004, which are hereby incorporated by reference herein.

1. An image motion detection apparatus comprising: an image displaydevice which displays a moving image; a period designation device whichdesignates, in accordance with user operation, a first period in whichmotion is present in the moving image displayed by said image displaydevice and a second period in which no motion is present; an upper/lowerlimit value determination device which determines an upper limit valueand a lower limit value of a threshold value on the basis of a changeamount in the moving image during the first period with respect to apredetermined reference image and a change amount in the moving imageduring the second period with respect to the predetermined referenceimage; a threshold value determination device which determines anarbitrary threshold value in accordance with user operation within arange from the upper limit value to the lower limit value of thethreshold value, which are determined by said upper/lower limit valuedetermination device; and a motion detection device which detects motionin the moving image on the basis of the threshold value determined bysaid threshold value determination device and the change amount in themoving image with respect to the predetermined reference image.
 2. Theapparatus according to claim 1, further comprising a change amount shiftdisplay device which displays a shift situation of the change amount inthe moving image with respect to the predetermined reference image. 3.The apparatus according to claim 2, further comprising a period changedevice which changes at least one of the first period and the secondperiod in accordance with user operation near the shift situation of thechange amount which is displayed by said change amount shift displaydevice.
 4. The apparatus according to claim 1, further comprising adetection target region setting device which sets an image region as amotion detection target of said motion detection device on the basis ofthe moving image during the first period.
 5. The apparatus according toclaim 4, wherein said detection target region setting device sets animage region in which motion is detected from the moving image duringthe first period as the image region as the motion detection target ofsaid motion detection device.
 6. The apparatus according to claim 1,further comprising a threshold value determination range display devicewhich displays the upper limit value and the lower limit valuedetermined by said upper/lower limit value determination device.
 7. Theapparatus according to claim 1, wherein said motion detection devicedetects the motion in the moving image when the change amount in themoving image with respect to the predetermined reference image is notless than the threshold value or when the change amount in the movingimage with respect to the predetermined reference image exceeds thethreshold value.
 8. The apparatus according to claim 1, furthercomprising a representative image display device which displays arepresentative image during at least one of the first period and thesecond period.
 9. The apparatus according to claim 1, wherein saidperiod designation device designates the first period and the secondperiod from a live image.
 10. The apparatus according to claim 1,wherein said period designation device designates the first period andthe second period from the moving image accumulated by said accumulationdevice.
 11. An image motion detection method by an image motiondetection apparatus which detects motion in a moving image, comprising:an image display step of displaying a moving image; a period designationstep of designating, in accordance with user operation, a first periodin which motion is present in the displayed moving image and a secondperiod in which no motion is present; an upper/lower limit valuedetermination step of determining an upper limit value and a lower limitvalue of a threshold value on the basis of a change amount in the movingimage during the first period with respect to a predetermined referenceimage and a change amount in the moving image during the second periodwith respect to the predetermined reference image; a threshold valuedetermination step of determining an arbitrary threshold value inaccordance with user operation within a range from the upper limit valueto the lower limit value of the threshold value, which are determined;and a motion detection step of detecting motion in the moving image onthe basis of the determined threshold value and the change amount in themoving image with respect to the predetermined reference image.
 12. Acomputer-readable storage medium which stores a program which causes acomputer to execute an image motion detection method by an image motiondetection apparatus which detects motion in a moving image, the storagemedium storing a program comprising program codes corresponding to animage display step of displaying a moving image; a period designationstep of designating, in accordance with user operation, a first periodin which motion is present in the displayed moving image and a secondperiod in which no motion is present; an upper/lower limit valuedetermination step of determining an upper limit value and a lower limitvalue of a threshold value on the basis of a change amount in the movingimage during the first period with respect to a predetermined referenceimage and a change amount in the moving image during the second periodwith respect to the predetermined reference image; a threshold valuedetermination step of determining an arbitrary threshold value inaccordance with user operation within a range from the upper limit valueto the lower limit value of the threshold value, which are determined;and a motion detection step of detecting motion in the moving image onthe basis of the determined threshold value and the change amount in themoving image with respect to the predetermined reference image.
 13. Animage motion detection apparatus comprising: an image display devicewhich displays a moving image; a start instruction device whichinstructs a start of automatic setting of sensitivity; and a thresholdvalue determination device which obtains, on the basis of a plurality offrames, a change amount in the moving image with respect to apredetermined reference image in a state in which the image is assumedto have no motion in response to the instruction of automatic setting bysaid start instruction device and determines a threshold value formotion detection from the obtained change amount, wherein said thresholdvalue determination device obtains, as the threshold value, a valueobtained by calculating an average value of different value sets of theframes except a predetermined amount on upper and lower sides and addinga predetermined offset to an average difference value of average valuesof the difference values of the plurality of frames except apredetermined amount on upper and lower sides.
 14. The apparatusaccording to claim 13, further comprising an interrupt device whichinterrupt automatic sensitivity setting when a camera parameter changesduring automatic sensitivity setting, and an interrupt display devicewhich displays the interrupt.
 15. The apparatus according to claim 14,wherein the camera parameter is at least one of a camera photographingposition, pan, tilt, zoom, focus, aperture, exposure, shutter speed, andwhite balance.
 16. An image motion detection apparatus comprising: animage display device which displays a moving image; a start instructiondevice which instructs a start of automatic setting of sensitivity; anda threshold value determination device which obtains, on the basis of aplurality of frames, a change amount in the moving image with respect toa predetermined reference image in a state in which the image is assumedto have no motion in response to the instruction of automatic setting bysaid start instruction device and determines a threshold value formotion detection from the obtained change amount, further comprising athreshold value setting device which displays sensitivity which isinversely proportional to the threshold value, and indirectly sets thethreshold value by causing a user to set the sensitivity.
 17. Theapparatus according to claim 16, further comprising a sensitivitysetting device which reflects a result of automatic sensitivity settingon the sensitivity display.